Motion robust vital signal monitoring

ABSTRACT

The present invention relates to a device and a method for extracting physiological information from remotely detected electromagnetic radiation emitted or reflected by a subject ( 10 ). A data stream ( 30 ) derivable from electromagnetic radiation ( 20 ) emitted or reflected by a subject ( 10 ) is received. The data stream ( 30 ) comprises a sequence ( 76, 152 ) of signal samples ( 78   a,    78   b,    78   c ) including physiological information and indicative of disturbing motion. The signal samples ( 78   a,    78   b,    78   c ) represent at least one region of interest ( 68 ) exhibiting an at least partially periodic indicative pattern attributable to at least one physiological parameter ( 56 ), and a non-indicative motion region ( 110 ). The sequence ( 76, 152 ) of signal samples ( 78   a,    78   b,    78   c ) is processed, comprising deriving a sequence ( 106, 158 ) of derivative motion compensated samples ( 108   a,    108   b,    108   c ) at least partially compensated for undesired overall motion; detecting an evaluation parameter representative of motion compensation accuracy; and deriving at least one characteristic signal ( 128, 172, 176 ) at least partially indicative of the at least partially periodic indicative pattern from the sequence ( 106, 158 ) of motion compensated samples ( 108   a,    108   b,    108   c ), wherein deriving the characteristic signal ( 128, 172, 176 ) is performed depending on the detected evaluation parameter.

FIELD OF THE INVENTION

The present invention relates to a device and a method for extractingphysiological information from remotely detected electromagneticradiation emitted or reflected by a subject, wherein the physiologicalinformation is embodied in a data stream comprising a sequence of signalsamples including physiological information and indicative of disturbingmotion.

BACKGROUND OF THE INVENTION

US 2010/0061596 A1 discloses a method of determining a similarity with aportion of a physiological motion, the method comprising the steps of:

-   -   obtaining a first image of an object;    -   obtaining a second image of the object;    -   determining a level of similarity between the first and the        second image; and    -   correlating the determined level of similarity between the first        and second images with a portion of the physiological motion.

The document further discloses several refinements of the method. Inparticular, the document is directed to patient monitoring, such asmonitoring breathing activity of a patient. Vital signal monitoringgrows in significance in several fields of application, such as patientmonitoring and monitoring sports and fitness activities, for example.Further beneficial applications can be envisaged. Although considerableprogress in the field of monitoring performance has been achieved, it isstill a challenge to provide for instant signal recognition and signalprocessing enabling immediate, so-to-say, on-line detection of desiredvital signals. This applies in particular to hand-held mobile devicescommonly lacking of sufficient computing power and typically exposed tochallenging monitoring conditions and constraints.

A further challenge may arise from disturbances and restrictions whichhave to be taken into account for the detection of the desired signals.As known in the art, detection quality can be improved through applyingobtrusive (or: tactile) measurement. For monitoring breathing activityor, in other words, respiration activity, obtrusive measurement devicesmay comprise belts or sensors which typically have to be attached to asubject's body. Furthermore, referring to remote detection approaches,prior art devices and methods may require markers or similar items whichhave to be applied to the subject to be observed. These markers can beremotely monitored since they provide sufficient “detectability” and maybe considered prominent targets for a detecting device. Still, however,obtrusive measurement, either applied remotely or via tactilemeasurement devices, is considered unpleasant and uncomfortable by manyobserved subjects.

Remote unobtrusive measurement typically enables a recording ormonitoring of the subject of interest without applying any components or“hardware” to the subject at all. Consequently, since no hardwaremarkers are available, remote unobtrusive detection is widely subjectedto disturbances. Recently, even mobile hand-held devices for remotemonitoring of vital signals have been envisaged. Mobile hand-helddevices are even more susceptible to disturbances since they aretypically hand-operated without fixed support. Consequently, hugedisturbances attributable to non-indicative device motion with respectto the subject to be monitored have to be expected.

Therefore, it has to be taken into account that the recorded data, suchas captured reflective or emitted electromagnetic radiation (e.g.,recorded image frames), typically comprises major signal componentsderiving from overall disturbances. Disturbance-related signalcomponents overlay and affect the desired vital signals basicallyaddressed when monitoring the subject. Overall disturbances may beattributed to changing luminance conditions and disturbing motioncomponents, for example. Disturbing motion may arise from non-indicativemotion of the subject itself, or from undesired motion of the detectingor sensing device. In particular with mobile hand-held monitoringdevices, overall motion (or: global motion) is considered a hugechallenge. Furthermore, particularly addressing respiration detectionvia remote unobtrusive measurement devices, subject motion-relatedsignals are, so-to-say, attenuated in case the subject of interest iscovered (at least partially), for instance, by clothes or even blankets.This applies in particular when sleeping or lying subjects areaddressed. Under such conditions, even removal of a blanket forimproving detection accuracy would be considered an unpleasant obtrusivemeasure. After all, vital signal detection becomes even more difficultsince amplitudes and/or nominal values of disturbing signal componentsare expected to be much larger than amplitudes and/or nominal values ofdesired signal components to be extracted. Potentially the magnitude ofdifference between the respective components (e.g., global motion vs.respiration motion) can be expected to even comprise several orders.

A possible approach to this challenge may be directed to providingwell-prepared and steady ambient conditions when capturing a signal ofinterest in which the desired vital signal component is embedded. Aminimization of potentially occurring disturbing signal components canbe achieved in this way. However, such “laboratory” conditions cannot betransferred into everyday field applications and environments since highefforts and much preparation work would be required therefore.

The required preparation work might comprise, for instance, installationand orientation of several defined standard light sources and, moreover,measures for fixation of the subject to be observed and of themonitoring device so as to avoid disturbing motion. It is consideredunlikely that these measures are applicable in everyday environments,such as ambulant or clinical patient monitoring, sleep monitoring, oreven in lifestyle environments like sporting and fitness monitoring.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a device and amethod for extracting physiological information from remotely detectedelectromagnetic radiation providing further refinements facilitatingobtainment of the desired vital signals with reduced efforts. It wouldbe further advantageous to provide a device and a method adapted forbeing less susceptible to disturbances, in particular to disturbancesarising from global motion artifacts. Furthermore, it would beadvantageous to provide for signal detection approaches enablingimproved detection accuracy and reliability.

In a first aspect of the present invention a device for extractingphysiological information from remotely detected electromagneticradiation emitted or reflected by a subject is presented, the devicecomprising:

-   -   an interface for receiving a data stream derivable from        electromagnetic radiation emitted or reflected by a subject, the        data stream comprising a sequence of signal samples including        physiological information, the signal samples being further        indicative of disturbing motion, the signal samples representing        at least one region of interest exhibiting an at least partially        periodic indicative pattern attributable to at least one        physiological parameter, and a non-indicative motion region; and    -   a processing unit configured for processing the sequence of        signal samples, comprising:        -   a stabilizing means configured for deriving a sequence of            derivative motion compensated samples at least partially            compensated for undesired overall motion;        -   a motion compensation assessment means configured for            detecting an evaluation parameter representative of motion            compensation accuracy; and        -   an extractor means configured for selectively deriving at            least one characteristic signal at least partially            indicative of the at least partially periodic indicative            pattern from the sequence of motion compensated samples,            wherein deriving the characteristic signal is performed            depending on the detected evaluation parameter.

The present invention is based on the insight that, even though motioncompensation measures can be applied to the detected signal samples, theresulting “motion compensated” samples can still be heavily affected bynon-indicative motion disturbance remainders. In such a case,advantageously, the respective “motion compensated” samples should beexcluded from downstream extraction and analyzation measures since thedesired vital signals of interest basically cannot be detected at arequired accuracy level.

In this way, it can be achieved that heavily motion corrupted “motioncompensation” samples cannot be reflected in correspondingly corruptedvital signals obtained through subsequent processing measures. Vitalsignals are based on the physiological information that is included inthe signal samples.

By contrast, alternative approaches are typically directed to assesssignal processing accuracy and, consequently, motion compensationaccuracy, at the level of the eventually processed and derived vitalsignals of interest. This can be considered disadvantageous since inthis way also heavily distorted samples can pass accuracy assessment andgraded as “good” samples, given that they accidentally fall within adefined value range assigned to “good” samples.

Compared with this, the present invention provides for a motioncompensation quality assessment sub-routine upstream of (or: proceeding)the actual vital signal extraction and analyzation measures. In otherwords, the accuracy assessment sub-routine can be interposed betweenmotion compensation and signal extraction measures. In this way,stability assessment can be applied to the motion compensated samplesprior to further proceeding.

When addressing remote signal detection on the basis of mobile orhand-held detection devices, often huge motion-related disturbancesnon-indicative of the signals of interest are present in the detecteddata. In other words, the presence of huge distortions has to be“accepted” and dealt with. Moreover, it has to be “accepted” that inmany environments and fields of application, depending on actualmonitoring conditions, at least part of the detected signal samples aredistorted to such an extent that the desired vital signals basicallycannot be derived therefrom. It is considered beneficial in such casesthat these heavily corrupted samples are excluded from furtherprocessing in case the motion compensation outcome has been evaluated asbeing insufficient. Consequently, as indicated above, the device of theinvention is particularly suited for remote signal detectionapplications intended for operation without obtrusive markers or similaritems. In this connection, mobile applications, hand-held devices, ormobile environments in general can be addressed.

In the field of remote vital signal detection, motion compensation isoften addressed. For instance, a transitional shift between two or moreconsecutive samples (or: image frames) can be estimated through adequateimaging processing algorithms. It goes without saying, that alsorotational or tilting movement can be addressed through adequatemeasures. Shift information or motion information can be utilized forcompensating the signals for undesired motion so as to “stabilize” thesequence. However, in this connection, it should be noted that thesequence of signal samples may for example comprise a considerableminute motion pattern of interest primarily addressed. Therefore,overall “smoothening” the sequence of image samples by directlytransferring commonly known image processing algorithms may rather levelthe signals and, consequently, remove the desired signal components fromthe signal samples.

It is therefore considered beneficial to exclude a highly-indicativeregion, the region of interest, from motion detection measures. It canbe achieved in this way that the minute motion pattern of interest isstill present (or: preserved) in the sequence of motion compensatedsamples. It should be further understood in this connection thattypically the characteristic signal which is considered indicative ofthe vital signal of interest is basically derived from the region ofinterest rather than from the non-indicative motion region which isbasically addressed for non-indicative motion detection.

The device of the invention is particularly suited for, but not limitedto, detecting a subject's respiration rate, respiration ratevariability, heart rate or related derivative parameters such as oxygensaturation. Occurrence and expectable characteristics of such vitalsignals can be readily predicted or assumed to a certain extent (e.g.,an assumed range of the respiration rate). Furthermore, for instance,when aiming at an extraction of the subject's present breath rate, itcan be assumed that a cycle of breathing in and breathing out isrepresented by a characteristic repetitive lifting and lowering of thechest portion and/or the abdominal portion of the subject's body.Needless to say, respiration can also be represented by a characteristicmotion of a face portion of the subject (e.g., nasal wings or mouthportions). Basically, indicative subject motion can be considered asphysiological information since it is representative of the underlyingdesired vital signals. In general, the term indicative motion patternmay refer to indicative subject motion-related characteristics (such asfrequency and/or amplitude) sought in the sequence of signal samples. Asused herein, the term sequence may refer to a continuous or discreteseries of signal samples.

The data stream may comprise a sequence of frames or, more precisely, aseries of image frames. For instance, RGB-images comprising colorinformation can be utilized. However, also frames representing infrared(IR) and red (R) information can form the sequence of frames. The imageframes can represent at least a portion of the observed subject andfurther elements. Typically, a frame may comprise a two-dimensionalarray of pixels. However, in some embodiments, a frame may comprise aline array, that is, a single line of pixels, for instance.

There exist several embodiments of the stabilizing means, the motioncompensation assessment means and the extractor means. In a first,fairly simple embodiment, the stabilizing means, the motion compensationassessment means and the extractor means are commonly embodied by theprocessing unit which is driven (or: controlled) by respective logiccommands (or: program code). Such a processing unit may also comprisesuitable input and output interfaces and, furthermore, additionalprocessing means. However, in the alternative, each or at least some ofthe stabilizing means, the motion compensation assessment means, theextractor means and (if any) further processing means can be embodied byseparate processing means which are controlled or controllable byrespective logic commands. Hence, each respective processing means canbe adapted to its special purpose. Consequently, a distribution of taskscan be applied, where distinct tasks are processed (or: executed) ondistinct single processors of a multi-processor processing unit, orwherein image processing-related tasks are executed on an imageprocessor, while other operational tasks are executed on a centralprocessing unit, for example. In a further embodiment the stabilizingmeans further comprises optical stabilization included in the device forextracting physiological information, which stabilizes image samples byvarying the optical path to the sensor means. The stabilizing means maybe implemented in the lens, e.g. using a floating lens element that ismoved orthogonally to the optical axis of the lens, or by moving thesensor means to counteract the movement of the device. In a furtherembodiment the device includes a movement sensor (e.g. an accelerometeror gyroscope) to detect device motion and to use detected motion tostabilize recorded image samples. The motion compensation assessment mayassess the quality of the motion compensation for example by detectingthe similarity between motion compensated image samples. Differentillustrative embodiments can take the form of entirely hardwareembodiments, entirely software embodiments, or of embodiments containingboth hardware and software elements. Some embodiments or aspects can beimplemented in software or program code. Program code may have the formof application software program code or of firmware program code, forexample.

According to an advantageous embodiment, the processing unit furthercomprises an analyzing means configured for determining temporalvariations in the characteristic signal, the temporal variations beingrepresentative of at least one vital signal.

Basically, the characteristic signal can be considered indicative orrepresentative of the at least one physiological parameter and, atleast, in a mediate way, of the desired vital signal of interest. Thismay apply to the subject's respiration rate, respiration ratevariability, pulse rate, blood pressure, heart rate, heart ratevariability, oxygen saturation (SpO2), perfusion index, or to respectivederivates such as a PPG (photo-plethysmography) wave signal that isderived from the pulsation of the arteries, which are all examples ofvital signals. Preferably, the desired vital signal is clearlydetectable in the at least one characteristic signal. Signal processingmethods can be utilized for extracting the desired signal.

As indicated above, since an upstream motion compensation assessmentprocedure is implemented, and since the characteristic signal is derivedunder consideration of the detected evaluation parameter, basically“good” signal samples are processed and, consequently, a considerably“clean” characteristic signal can be delivered to the analyzing means.It should be understood that the characteristic signal can still besomewhat distorted due to a variety of disturbances. However, it isemphasized that the characteristic signal derived under consideration ofthe detected evaluation parameter may have a significantly improvedsignal-to-noise ratio since heavily distortion affected motioncompensated samples are removed from the signal basis upon which thecharacteristic signal can be determined. As indicated above, vitalsignal extraction and analyzation is primarily directed to the region ofinterest in the signal samples and the motion compensated samples,respectively.

According to yet another embodiment the evaluation parameter is a flagparameter representative of a state of a set of states indicative ofmotion compensation accuracy for a given motion compensated sample, or agiven set of motion compensated samples. It should be understood in thisconnection that motion compensation assessment can be directed to thesingle sample level or to a set of a plurality of samples. It can beenvisaged in the latter case that the evaluation parameter is a movingaverage evaluation parameter spanning over a plurality of motioncompensated samples.

In some embodiments, two states can be assigned to the flag-likeevaluation parameter. The states may represent “good” motion compensatedsamples and “bad” motion compensated samples. Typically, good samplescan be utilized during further processing measures. By contrast, badsamples can be excluded from further processing measures. For the sakeof understanding, the states can also be regarded as color-coded. Agreen flag may represent a good motion compensated sample. A red flagmay represent a bad motion compensated sample. For grading or evaluatingthe motion compensated samples (or the sets of motion compensatedsamples), threshold values can be defined. The threshold values canrepresent motion compensation accuracy-related parameters utilized bythe motion compensated assessment means.

According to a further alternative embodiment, the flag-like evaluationparameter can be representative of more than two motion compensationaccuracy-indicative states. For instance, three different states mayform the set of states. By way of example, a “good” (or: green) flag maybe assigned to clearly good motion compensated samples. Furthermore, a“bad” (or: red) flag may be assigned to bad motion compensated samples.In order to expand the set of states, an “average” (or: yellow) flag maybe assigned to motion compensated signal samples (or: respective sets)which have been evaluated as providing medium motion compensationquality. Of course, even further intermediate states can be envisaged.By providing a set of three states, a border area can be indicated inwhich the risk of occurrences of bad motion compensated samples ispresent. It should be understood in this connection that, in oneembodiment, yellow flagged (average) samples are still considered to beapplicable for further signal processing measures. A set of statesproviding more than a good state and a bad state enables to drawattention to a transition area between corrupted samples andhigh-quality samples. In this way, for example, a user can be advised tokeep the monitoring device stable so as to keep away from the bad state.

In case even further states form the set of states a user feedback canbe even further detailed. For instance, in case four states areutilized, a first state may be assigned to clearly corrupted samples. Asecond state may be assigned to samples which are still consideredinapplicable but providing motion compensation accuracy close to thethreshold. A third state can be assigned to samples which are consideredto be applicable but also close to the threshold value for motioncompensation accuracy. A fourth state may be assigned to samples whichare high graded samples comprising a significantly high signal-to-noiseratio.

According to yet an even further embodiment, the extractor means, on thebasis of the actual evaluation parameter, selectively performs or omitsprocessing the respective motion compensated samples for deriving the atleast one characteristic signal. Corrupted samples or sets of samplescan be excluded from the vital signal detection in this way. It shouldbe further noted that the evaluation parameter can be a discreteparameter representing discrete values or flags. However, in thealternative, the evaluation parameter can also be defined as a decimalparameter which may represent decimal numbers and, consequently, evenfurther intermediate values enabling to describe motion compensationaccuracy with precise figures.

According to still yet another embodiment, the stabilizing means isconfigured for deriving the sequence of derivative motion compensatedsamples under consideration of at least one portion of thenon-indicative motion region in the signal samples.

As indicated above, it is preferred that motion estimation or motiondetermination which may provide input parameters for motion compensationis not based on the region of interest. In particular, for hand-held ormobile devices, sensor motion (or: camera motion) with respect to thesubject of interest can be addressed in this way. It is thereforepreferred that the at least one portion of the non-indicative motionregion which is utilized for motion compensation represents considerablystatic elements or objects which may serve as a proper basis for motionestimation.

According to a further aspect of this embodiment, the stabilizing meansis further configured for detecting and tracking local features in theleast one portion of the non-indicative motion region in the signalsamples of the sequence. To this end, in one embodiment, optical flowconsiderations can be utilized. In this connection, the stabilizingmeans can make use of Lucas-Kanade-tracking approaches. In thealternative, or in addition, feature-based image registration approachescan be utilized. Furthermore, additionally or alternatively, featuredetection approaches can be exploited, such as edge detection, cornerdetection, and blob detection, respectively. Still, according to analternative or additional aspect, feature description models can be usedfor feature tracking and, consequently, for motion estimation. Featuredescription models may comprise scale-invariant feature transformation(SIFT), speed-up robust feature detection (SURF), gradient location andorientation histogram image description (GLOH), histogram of orientedgradients feature description (HOG), local energy based shape histogramimage description (LESH), etc. Again, it is worth noting that theafore-mentioned approaches are preferably applied to the at least oneportion of the non-indicative motion region since in this way undesiredexaggerated motion compensation within the region of interest and,therefore, adversely affecting the desired indicative motion pattern canbe avoided. It is worth noting in this connection that indeed motioncompensation measures can be applied to the region of interest. However,given that motion detection is performed outside the region of interest,rather global motion compensation instead of indicative motioncompensation is applied to the region of interest. Typically, the subtlemotion pattern of interest can be preserved in this way.

According to another aspect of the device, the motion compensationassessment means is configured for detecting a similarity between motioncompensated samples under consideration of at least one portion of thenon-indicative motion region in the motion compensated samples.

Also in this connection it is preferred in one embodiment that motioncompensation assessment is not performed in the region of interest. Byway of example, motion compensation assessment can be based on areference sample which may serve as a reference to which an actualmotion compensated sample is compared. The reference sample can be afixed reference sample, for instance, an initial sample out of asequence of samples. In the alternative, the reference sample can be amoving reference sample, wherein a defined (temporal) distance orrelation between the reference sample and the actual motion compensatedsample is kept when a series of motion compensated samples is processedfor motion compensation assessment measures. The reference sample andthe actual motion compensated sample can be consecutive samples orsamples that are spaced in the sequence of motion compensated samples,that is, intervening samples may be present.

As indicated above, the evaluation parameter can be based on processinga single motion compensated sample (and a respective reference sample).In the alternative, the evaluation parameter can represent motioncompensation accuracy of a set of motion compensated samples withrespect to respective reference samples. In this way, the evaluationparameter may represent a moving average value for motion compensationaccuracy.

As used herein, the term non-indicative motion region may basicallyrefer to a portion of both the (input) signal samples and the motioncompensated samples which is not occupied by the region of interest.Consequently, the terms region of interest and non-indicative motionregion can be used in connection with both the initially motion-affectedsignal samples and the motion compensated samples. The term “at least aportion of the non-indicative motion” region may refer to a respectivesubset.

According to a further aspect of the above embodiment, the motioncompensation assessment means is further configured for applying anabsolute difference processing algorithm to the at least one portion ofthe non-indicative motion region in a respective motion compensatedsample with respect to a reference sample. By way of example, the sum ofabsolute differences (SAD) approach can be chosen so as to detect asimilarity between actual motion compensated samples and respectivereference samples. A remaining difference between respective processedsamples may be reflected in the evaluation parameter.

According to another aspect, the motion compensation assessment means isconfigured for detecting feature correspondences in at least one portionof the non-indicative motion region in a respective motion compensatedsample and in a reference sample. It is worth noting again thatbasically feature detection is not directed to the region of interestbut rather to the non-indicative motion region. By way of example, anumber of detected feature correspondences can indicate a degree ofmotion compensation accuracy and, consequently, be reflected in thedetected evaluation parameter. As already indicated above, the referencesample can be formed by a fixed reference sample or by a movingreference sample keeping a defined relation (distance or gap) to acurrently processed motion compensated sample.

According to yet another preferred embodiment, the motion compensationassessment means is further configured for detecting the evaluationparameter under consideration of a plurality of motion compensationassessment indicators. By way of example, motion compensation can makeuse of both absolute difference processing and feature correspondencesdetecting applied to the motion compensated samples under considerationof respective reference samples. In this way, at least two motioncompensation accuracy-related indicators can be obtained. It can betherefore defined that certain conditions have to be met so as toselectively perform or omit further processing of the respective motioncompensated sample (or the respective set of samples) for deriving theat least one characteristic signal. In this connection, two respectivethresholds can be defined such that further processing of the motioncompensated samples is performed merely in case both indicators areabove (or, respectively, below) the thresholds differentiating themotion compensated samples into good ones and bad ones. However in thealternative, the device can be configured such that further processingof the respective motion compensated samples is conducted in the eventthat at least one indicator exceeds (or comes below) the respectivethreshold.

According to yet a further aspect, the device further comprises a signalgeneration unit configured for generating a noticeable output signaldepending on the actual evaluation parameter, wherein the output signalpreferably indicates a state of a set of states indicative of motioncompensation accuracy. In this way, the device can further provide userfeedback and advice directed to further enhance motion compensationaccuracy and, consequently, signal detection accuracy. As used herein,the term noticeable output signal may relate to an output signal whichis clearly noticeable for a user of the device. The output signal maycomprise, for example, sound signals, speech, visible signals,indicating lights, displayed information, tactile signals, and suitablecombinations thereof. The output signal or the output signals mayrepresent the actual evaluation parameter. Therefore, for instance, thesignal generation unit can be configured for displaying or transmittingred light (representing bad samples), green light (representing greensamples) and, if any, yellow light (representing average samples).Noticing a red indicating light, the user may be advised to keep themonitoring device stable since currently no vital signal processing canbe conducted due to motion-related disturbances. Noticing a greenindicating light, the user can be informed that the device is workingwell and that signal detection accuracy clearly exceeds a thresholdlevel. Noticing a yellow indicating light, the user can be informed thatcurrently signal processing can be conducted but that signal processaccuracy (or, more precisely, motion compensation accuracy) is close tothe threshold level. In this way, the user can be prompted to reduceadverse motion influences.

According to yet another embodiment, the device further comprises asensor means, particularly a hand-held sensor means, configured forcapturing electromagnetic radiation within at least one particularwavelength range selected from the group consisting of visible light,infrared light, and ultraviolet radiation, the sensor means beingconnectable to the interface. By way of example, the sensor means can beembodied by a camera. For instance, the sensor means can be embodied bya mobile device having an integrated camera or an attachment camera,such as a personal digital assistant, a mobile phone, a mobile computer,a tablet computer, etc. However, in the alternative, the sensor meansalso can be embodied by a laser scanner device. Furthermore, the sensormeans can be part of a mobile medical monitoring device.

According to another aspect, the device may further comprise at leastone source of illumination configured for emitting radiation. In thealternative, the device can also make use of external or even ambientradiation sources. According to another aspect, the analyzing means isfurther configured for applying an integral transformation, for examplea Fourier transformation to the at least one characteristic signal,thereby obtaining frequency information attributable to the desiredindicative subject motion pattern representative of the vital signal.

In a further aspect of the present invention, a method for extractingphysiological information from remotely detected electromagneticradiation emitted or reflected by a subject is presented, the methodcomprising the steps of:

-   -   receiving a data stream derivable from electromagnetic radiation        emitted or reflected by a subject, the data stream comprising a        sequence of signal samples including physiological information        and indicative of disturbing motion, the signal samples        representing at least one region of interest exhibiting an at        least partially periodic indicative pattern attributable to at        least one physiological parameter, and a non-indicative motion        region; and    -   processing the sequence of signal samples, comprising:    -   deriving a sequence of derivative motion compensated samples at        least partially compensated for undesired overall motion;    -   detecting an evaluation parameter representative of motion        compensation accuracy; and    -   deriving at least one characteristic signal at least partially        indicative of the at least partially periodic indicative pattern        from the sequence of motion compensated samples, wherein        deriving the characteristic signal is performed depending on the        detected evaluation parameter.

Advantageously, the method can be carried out utilizing the device forextracting physiological information of the invention.

According to an embodiment the method further comprises the steps of:

-   -   on the basis of the actual evaluation parameter, selectively        performing or omitting processing the respective motion        compensated samples for deriving the at least one characteristic        signal; and    -   determining temporal variations in the characteristic signal,        the temporal variations being representative of at least one        vital signal.

In yet another aspect of the present invention, there is provided acomputer program which comprises program code means for causing acomputer to carry out the steps of the processing method when saidcomputer program is carried out on the computer. The computer programcomprises program code means for causing a computer to carry out thesteps of the method as claimed in claim 13 or 14 when said computerprogram is carried out on the computer.

As used herein, the term computer stands for a large variety ofprocessing devices. In other words, also mobile devices having aconsiderable computing capacity can be referred to as computing device,even though they provide less processing power resources than standarddesktop computers. Furthermore, the term “computer” may also refer to adistributed computing system which may involve or make use of computingcapacity provided in a cloud environment.

Preferred embodiments of the invention are defined in the dependentclaims. It shall be understood that the claimed methods and the claimedcomputer program can have similar preferred embodiments as the claimeddevice and as defined in the dependent device claims.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention will be apparent from andelucidated with reference to the embodiments described hereinafter. Inthe following drawings:

FIG. 1 shows a schematic illustration of subject motion indicative of anexemplary vital signal;

FIG. 2 shows a schematic illustration of a general layout of a device inwhich the present invention can be used;

FIG. 3 shows a schematic illustration of an arrangement including asubject to be monitored;

FIG. 4 shows an exemplary simplified illustration of a mobile device inwhich the present invention can be used;

FIG. 5 shows a simplified exemplary sequence of signal samples which aresubjected to overall motion;

FIG. 6 illustrates a simplified block diagram representing several stepsof an embodiment of a method in accordance with the invention;

FIG. 7 a illustrates an exemplary motion compensation accuracyassessment approach;

FIG. 7 b illustrates an alternative exemplary motion compensationaccuracy assessment approach;

FIG. 8 illustrates yet another alternative exemplary motion compensationaccuracy assessment approach;

FIG. 9 a exemplifies an illustration of a characteristic signal obtainedfrom motion corrupted samples;

FIG. 9 b exemplifies an illustration of another characteristic signalobtained from motion compensated samples classified as good samples; and

FIG. 10 shows an illustrative block diagram representing several stepsof an embodiment of a method according to the invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a schematic illustration of a subject 10 which experiencesmotion indicative of a signal of interest. The subject 10 undergoes acharacteristic motion of an indicative portion 12 due to respiration.When breathing, expansion and contraction of the lungs or the diaphragmcauses slight motion of characteristic portions in living beings, inparticular lifting and lowering of the chest. Also abdominal breathingcan cause characteristic motion of respective parts of the subject'sbody. At least partially periodic motion patterns induced by variousphysiological processes can occurred in many living beings, particularlyin humans or animals. Over time, as indicated by an arrow 16, theindicative portion 12 is moved between a contracted position, indicatedby reference numerals 12 a, 12 c, and an extracted position, indicatedby reference numerals 12 b. By way of the example, based on this motionpattern (herein also referred to as physiological information 56, referto FIG. 2) the respiration rate or respiration rate variability can beassessed. While the indicative portion 12 is pulsating over time, anon-indicative portion 14 remains substantially motionless (in terms ofthe desired motion pattern). Certainly, also the non-indicative portion14 can undergo diverse motion over time. However, this motion typicallydoes not correspond to the periodic pulsation of the indicative portion.

In another example the characteristic movement of the indicative portion12 results from pulsating arteries in the skin of the subject. Thepulsating arteries cause a minute motion of the surface texture due tothe pumping of the heart.

Now referring to FIG. 2, a device for extracting information isillustrated and denoted by a reference numeral 18. The device 18 isparticularly suited for detecting motion of an indicative portion 12 ofthe subject 10 which is related to physiological information, refer alsoto the arrow 56 in this regard. The device 18 can be utilized forrecording image frames representing the subject 10. The image frames canbe derived from electromagnetic radiation 20 emitted or reflected by thesubject 10. For extracting information from the recorded data, e.g. froma sequence of image frames, a defined part or portion of the subject 10can be observed by a sensor means 22. The sensor means 22 can beembodied, for instance, by a camera adapted to capture informationbelonging to at least one respective component of the electromagneticradiation 20. The sensor means 22 may comprise an array of single sensorelements. For instance, the sensor means 22 can make use of a line arrayor of a matrix array of single sensors, such as charge-coupled devices(CCD-sensors). Still, however, also alternative sensor types can beutilized. It is worth noting that the device 18 also can be primarilyconfigured for processing input signals, namely an input data stream,already recorded in advance, and, in the meantime, stored or buffered.In this connection, recording can be performed by a separate remotesensor means.

As indicated above, the electromagnetic radiation 20 can contain acontinuous or characteristic signal which is considered to be highlyindicative of at least one at least partially periodic vital signal but,on the other hand, typically massively distorted by overall disturbancessuch as global motion and varying illumination conditions. This appliesin particular when the device 18 or, at least, the sensor means 22 isarranged as a remote mobile device. In some embodiments, the device 18can make use of defined illumination sources or, in general, radiationsources 24, 28. Illumination source 24 can be considered an ambientseparate source of radiation. Illumination source 28 can be consideredan internal controllable source of radiation. The radiation sources 24,28 basically emit incident radiation 26 a, 26 b striking the subject 10.Typically, the incident radiation 26 a, 26 b is at least partiallyreflected by the subject 10. Furthermore, in particular for embodimentsmaking use of infrared (e.g., near-infrared or deep-infrared) radiation,also the subject 10 may emit (or: generate) radiation portions, such asthermal radiation.

Known methods for obtaining vital signals such as respiration-relatedsignals comprise tactile respiration rate monitoring and remoterespiration rate monitoring relying on markers applied to the subject ofthe interest. To this end, however, obtrusive monitoring is required. Asindicated above, an alternative approach is directed to remoteunobtrusive measuring utilizing specific image processing methods.

The sensor means 22 can be configured for delivering a data stream 30 toan interface 32. Needless to say, also a buffer means could beinterposed between the sensor means 22 and the interface 32. Downstreamof the interface 32 a stabilizing means 34 may be provided. Basically,the stabilizing means 34 can be configured for applying motioncompensation measures to the data stream 30. In this way, a sequence ofsignal samples embodied in the data stream 30 can be transformed into asequence of derivative motion compensated samples. At this level, motioncompensation is directed to overall motion which can be caused byrelative motion between the subject 10 and the sensor means 22. As usedherein, overall motion primarily relates to motion of the sensor means22 or, in general, motion of the device 18 comprising the sensor means22. A monitoring environment typically comprises at least a part of thesubject 10, peripheral elements such as walls, furniture or evennon-indicative portions of the subject 10, and the sensor means 22directed to the subject 10. Among these elements, undesired relativemotion can occur. In particular, for mobile applications making use ofhand-held portable devices 18 or, at least, sensor means 22, thedetected sequence of signal samples can be heavily affected by shakingor blurring effects caused on the end of the sensor means 22. Typically,these undesired disturbances exceed the desired patterns in magnitude.

These desired patterns may for example be motion patterns and forexample result from a desired subject movement such as a respiratoryinduced movement of the chest. The characteristic movement of theindicative portion 12 may for example also result from pulsatingarteries in the skin of the subject causing minute motion patterns. Theradiation sources 24, 28 (see FIG. 2) emit incident radiation 26 a, 26 bstriking the subject 10. By travelling through the skin the radiationundergoes an amount of absorption that depends on the length of the paththat the radiation travels through the skin and the absorptioncoefficient of the substance (e.g. blood, tissue). As the arteries arepulsating their diameter changes over time with the blood volume pulsecausing in the region of interest the intensity of the electromagneticradiation emitted or reflected by the subject to change with thefrequency of the heart rate.

In another example the desired patterns relate to patterns in frequencyand intensity of reflected or emitted light such as skin colorvariations. The pulsation of arterial blood causes changes in lightabsorption. Those changes form a PPG (photo-plethysmography) signal(also called, among other, a pleth wave). It is based on the principlethat temporal variations in blood volume in the skin lead to variationsin light absorptions by the skin. Such variations can be registered by avideo camera that takes images of a skin area, e.g. the face, whileprocessing calculates the pixel average over a selected region ofinterest (typically part of the cheek). By looking at periodicvariations of this average signal, the heart beat rate and respiratoryrate can be extracted. A method to measure skin colour variations,called Photo-Plethysmographic imaging (PPG), is described in WimVerkruysse, Lars O. Svaasand, and J. Stuart Nelson, “Remoteplethysmographic imaging using ambient light”, Optics Express, Vol. 16,No. 26, December 2008.

As indicated above, primary motion compensation measures may result insignal samples which may still contain motion-related disturbances.Therefore, depending on present motion influences affecting the signalsamples, in some cases also motion compensated samples can still beheavily distorted and therefore not applicable for further processingdirected to the extraction of the signal of interest. The presentembodiment basically tackles this issue.

The sequence of motion compensated samples can be delivered to a motioncompensation assessment means 36. The motion compensation assessmentmeans 36 can be configured for detecting an evaluation parameterrepresentative of motion compensation accuracy. For instance, the motioncompensation assessment means 36 can be adapted for determiningremaining motion-related distortions in the motion compensated signalsamples. In this way, an evaluation parameter can be obtained which isrepresentative of current motion compensation accuracy. The evaluationparameter can be a quality-related parameter. The evaluation parametercan be represented by a value on a scale having a certain range and,furthermore, a threshold value can be defined in this range fordetermining sufficient motion compensation accuracy and non-sufficientmotion compensation accuracy. Consequently, respective motioncompensation samples can be flagged so as to indicate whether they areconsidered applicable for further signal processing measures or stilldistorted in such a way that no further processing measures on the basisof these samples are recommended.

Consequently, these samples can be excluded from further processing. Insome embodiments, the evaluation parameter also can be configured as aflag parameter, wherein a flag can be assigned to signal samples (orrespective sets of signal samples), wherein the flag may represent astate of a set of distinct states. The group or set of states maycomprise at least one of a bad (or: red) state to be assigned to badsamples and a good (or: green) state to be assigned to good samples. Asindicated above, further intermediate stages can be considered.

The device 18 may further comprise an extractor means 38 configured forselectively deriving at least one characteristic signal at leastpartially indicative of the at least partially periodic indicativepattern from the sequence of motion compensated samples deliveredthereto. This periodic indicative pattern may be a motion patternresulting from a desired subject movement such as for example arespiratory induced movement of the chest or a pattern in frequency andintensity of the reflected or emitted light caused by changes in lightabsorption of the skin resulting from the pulsation of arterial blood inthe skin.

It is preferred that the extractor means 38 is configured for derivingthe characteristic signal under consideration on the detected evaluationparameter. In this way, “bad” motion compensated samples can be excludedfrom further processing. This may apply to a single bad motioncompensated sample or to a set of a plurality of bad motion compensatedsamples. In this way, signal derivation accuracy can be improved sincedistortions attributable to insufficient motion compensation accuracycan be prevented, at least to a certain extent. It should be noted inthis connection, that the sample pool or basis for the derivation of thecharacteristic signal can be reduced or thinned out in this way.Consequently, in particular when a set comprising a large quantity ofmotion compensated samples is excluded from further processing, the atleast one vital signal eventually cannot be determined for therespective period of time. However, it is considered advantageous toskip the characteristic signal derivation and the vital signaldetermination based thereon for corrupted (bad) motion compensatedsamples, compared to processing also corrupted samples without anyreflection or consideration of the potential outcome in respect of thedesired vital signal of interest.

It should be further noted that, given that only single or only a few ofcorrupted motion compensated samples are excluded from furtherprocessing, in some embodiments the characteristic signal still can bederived and established in a sufficient manner such that eventually thevital signal of interest can be extracted therefrom without considerableinterrupt. This may be the case in particular in environments wherein asample rate or frame rate in the input sequence is sufficiently high incomparison to a frequency, if any, of the vital signal of interest.

The device 18 may further comprise an analyzing means 40 configured fordetermining temporal variations of the characteristic signal. Inparticular, the analyzing means 40 can be adapted for seeking fordominant frequencies attributable to the vital signal of interest.Hence, the analyzing means 40 can make use of several signal processingapproaches. For instance, the analyzing means 40 can be configured forapplying, among other algorithms, a Fourier transformation or a similarintegral transformation to the characteristic signal so as to obtainfrequency values or even a frequency domain representation of theenhanced characteristic signal.

Eventually, a processed data stream 42 can be generated. The processeddata stream 42 can be delivered to an interface 44. Consequently, viathe interface 44, output data 46 can be made available to furtheranalysis and/or for display measures. The (input) interface 32 and the(output) interface 44 can be embodied by the same (hardware) interfaceelements. The stabilizing means 34, the motion compensation assessmentmeans 36, the extractor means 38 and (if any) the analyzing means 40 oreven further processing means can be embodied by a common processingunit 52. Also the interfaces 32, 44 can be connected thereto in a commonprocessing device accommodating the respective subcomponents. By way forexample, the processing unit 52 can be embodied by a personal computeror a mobile computing device.

Furthermore, the device 18 can comprise a signal generation unit 48which can be configured for generating an output signal which isnoticeable to a user of the device 18. It is preferred that the outputsignal is generated under consideration of the actual evaluationparameter detected by the motion compensation assessment means 36. Inother words, in some embodiments, the motion compensation assessmentmeans 36 can be utilized for “triggering” the signal generation unit 48.The signal generation unit 48 can indicate the actual evaluationparameter to a user of the device 18. As mentioned above, the evaluationparameter can be detected under consideration of a single sample or, inthe alternative, under consideration of a set of samples. Consequently,the signal generation unit 48 can be also adapted for representing amean evaluation parameter which may be a moving average evaluationparameter spanning over a plurality of samples. The signal generationunit 48 can make use of a single or a plurality of indicator sourcemeans 50. In this connection, the signal generation unit 48 shown inFIG. 2 comprises an exemplary, but non-limiting, set of indicatorsources 50 a, 50 b, 50 c. Indicator source 50 a can be embodied by avisual indicator source or light indicator source. For instance, theindicator source 50 a can comprise one or more light sources, such aslight emitting diodes (LED). Given that the evaluation parameter isrepresentative of several distinct states indicative of motioncompensation accuracy, each of the single light sources of the indicatorsource 50 a may represent a respective color. For instance, at least ared and at least a green LED may be utilized. In another embodiment, theindicator source 50 a comprises a source of red light, a source ofyellow light, and a source of green light. Needless to say, theindicator source 50 a can also be configured for cooperating with filtermeans such that different indicator lights can be generated underutilization of a single light source. In some embodiments, the visibleindicator source 50 a can make use of a display means. In thisconnection, LCD displays, LED displays and similar display types can beenvisaged. A display may present color information and/or textualinformation.

Additionally, or in the alternative, the signal generation unit 48 alsocan make use of a sound indicator source 50 b. The sound indicatorsource 50 b can comprise at least one sound generator, for instance, aloudspeaker. In some embodiments, the sound indicator source 50 b can beconfigured for presenting a speech message. However, in the alternative,also a single tone or a tone sequence can be generated by the soundindicator source 50 b.

According to another alternative embodiment, the signal generation unit48 can further comprise a tactile indicator source 50 c. By way ofexample, the tactile indicator source 50 c can be embodied by a buzzeror a vibration element. In this way, a subtle signal can be directed tothe user of the device 18. Each or at least some of the indicatorsources 50 a, 50 b, 50 c can be utilized for providing feedback to theuser. The feedback can be generated depending on the current evaluationparameter detected by the motion compensation assessment means.Dependent on the current motion compensation accuracy state, the usercan be assured that motion compensation measures are currentlyconsidered sufficient for enabling a proper vital signal extraction.However, in the alternative, the user feedback may also indicate thatmotion compensation measures are currently insufficient such that thedesired vital signal extraction is currently not enabled. Furthermore,the user can be advised to keep the device 18 or, at least, the sensormeans 22 stable so as to reduce overall motion influences. Needless tosay, further feedback messages can be directed to the user. In someembodiments, also the signal generation unit 48 can be accommodated orconnected to the processing unit 52.

In case also the sensor means 22 is jointly connected to the processingunit 52, a common housing may accommodate the respective components. Inthis connection, an overall system boundary is indicated by a referencenumeral 54. Reference numeral 54 may also refer to a common housing forthe device 18. If such an integrated approach is intended, the device 18can be embodied by a mobile device such as a smartphone, a tabletcomputing device or a mobile health monitoring device. These devices canmake use of an integrated sensor means (camera) 22 or, at least, beingconnectable to a separate sensor means (camera) 22. In another exemplaryconfiguration, the device 18 is a stationary device while at least thesensor means 22 is portable. The sensor means 22 can be coupled to astationary processing unit 52 via suitable cable connections or wirelessconnections.

With reference to FIG. 3, a common environment in which unobtrusivevital signal monitoring is performed is presented. The subject 10, e.g.a patient staying in bed, is resting on a support. The subject's 10 headwhich is attributable to the non-indicative portion 14 (FIG. 1) isexposed and pillowed, while the indicative portion 12, e.g., the chest,is covered by a blanket 64. Thus, the desired signal caused by a motionof the indicative portion 12 is attenuated or hidden. Therefore,obtrusive signal detection is considerably difficult. This applies inparticular when a portable mobile monitoring device 18 a is utilized.The monitoring device 18 a can comprise a handle 70 a user may grab forholding and orientating the device 18 a. The device 18 a, in particularthe sensor means 22, can be positioned and orientated such that theindicative portion 12 undergoing the indicative motion pattern can beobserved. In this connection, an exemplary region of interest 68representing the chest portion of the subject 10 is indicated by aquadrangular box. In FIG. 3, an axis 60 indicates an expected directionof the periodic motion pattern of interest. Periodic subject motionalong this axis 60 can represent the desired physiological information56 (FIG. 2). By contrast, potential subject motion in other directions,refer to reference numerals 62 a, 62 b, is considered to be notindicative and therefore not of particular interest. The monitoringenvironment shown in FIG. 3 may further comprise stationary objects,refer to reference numeral 66. Stationary objects 66 may serve asreference objects which may be utilized for motion compensation.Consequently, also the stationary object 66 (e.g., a chair) may bepresent in the field of view of the sensor means 22. For mobile orportable applications, the sensor means 22 may undergo positionalchanges and orientation changes when observing the subject 10.Typically, motion of the sensor means 22 may comprise motion along andaround several axes, refer to reference numerals 72 a, 72 b, 72 c.

FIG. 4 illustrates an alternative embodiment including a mobile device18 b. Since mobile devices such as mobile phones, tablet computers andnotebooks are readily available and, moreover, often include adequatecameras, suitable control algorithms can be implemented so as to controlthese devices in vital signal monitoring applications. The device 18 bmay comprise a display 74 for representing an indicative portion 12(represented by the region of interest 68) of the subject 10 exhibitingthe desired motion pattern which is attributable to the physiologicalinformation 56 of interest. A user may therefore target the subject 10under consideration of a present representation of the subject 10 in thedisplay 74. Consequently, instant signal detection on a remote basis canbe simplified. Needless to say, the device 18 b can also comprise thesignal generation unit 48 (FIG. 2) and at least one of the indicatorsources 50 a, 50 b, 50 c. To this end, available implemented signalsources can be utilized.

FIG. 5 illustrates a sequence 76 of signal samples 78 a, 78 b, 78 c. Forillustrative purposes, also a spatial reference 80 is indicated. Thesequence 76 may comprise a series of signal samples 78 a, 78 b, 78 c.Since motion-related disturbances are to be expected, typically a fieldof view covered by each of the signal samples 78 a, 78 b, 78 c may varyover the series of samples. Since these deviations are considered toexceed the desired indicative motion pattern in terms of absolute valuesand amplitudes, motion compensation is crucial for further processingand signal extraction. As already set out above, a sequence of motioncompensated samples 106 can be derived from the (original) sequence 76through motion compensation measures.

FIG. 6 illustrates a simplified exemplary flow chart diagramrepresenting a method in accordance with an embodiment of the invention.The flow chart basically describes a vital signal extraction processwhich can be applied to signal samples 78 a, 78 b, 78 c in a sequence76. Initially, the process may be started or triggered at operation 90.At operation 92, a to-be-processed signal sample (e.g., an image frame)is received. As already indicated above, signal processing can bedirected to single samples (out of a sequence) or to a set of samples.It should be understood in this connection that the process depicted inFIG. 6 can be understood as a “moving” process consecutively processingconsecutive entities in a series (or: sequence) of signal samples.

Subsequently, motion compensation processing (reference numeral 94) canbe applied to the to-be-processed sample. Basically, a motioncompensated sample can be obtained in this way. A motion compensationaccuracy assessment subroutine (process 96) may follow. The motioncompensation accuracy assessment subroutine can determine a currentmotion compensation accuracy level. In this way, an evaluation parametercan be determined which can be considered a motion compensation accuracyindicator value. Depending on whether the indicator value exceeds (orcomes below) a defined threshold, it can be decided whether or not thecurrently assessed motion compensated sample (or the currently assessedset of motion compensated samples) is to be considered during subsequentsignal extraction and processing operations. In case a desired accuracylevel is found to be met by the respective motion compensated samples,the process may proceed with operation 98 in which signal processing,for instance, respiration rate processing is conducted. In this way, thevital signals of interest can be extracted. Vital signal extraction maycomprise a derivation of characteristic signals from the sequence ofapproved proper motion compensated signal samples. Further signalprocessing algorithms may be involved. Subsequently, in a deliveryoperation 100, the vital signals of interest can be made available fordisplay measures, for data storage, and for further data processing.

In case it is found in the motion compensation accuracy assessmentoperation 96 that a desired accuracy level is not met by the processedmotion compensated sample, the respective sample can be excluded fromfurther processing, that is, for instance, from the operations 98 and100. In other words, the operations 98 and 100 can be bypassed. Instead,an alternative operation 102 may follow in which an output signal can begenerated and presented to a user pointing to that situation. Forinstance, the user can be advised to reduce adverse motion influences bykeeping the monitoring device stable. Consequently, an operation 104 mayfollow in which a next to-be-processed signal sample can be chosen.

It is worth mentioning in this connection that also in the event thatthe motion compensated signals are found to meet the desired accuracylevel, a respective output signal can be generated and presented to theuser, refer to the dashed line connected to output signal generationoperation 103. Regardless of the outcome of the motion compensationaccuracy assessment subroutine 96, eventually the operation 104 mayfollow in which the next to-be-processed signal sample can be chosen.Consequently, a plurality of signal samples in a sequence can beprocessed.

FIGS. 7 a, 7 b and FIG. 8 illustrate exemplary approaches to motioncompensation accuracy assessment. In FIG. 7 a, a motion compensatedsequence 106 of motion compensated samples 108 a, 108 b, 108 c is shown.Each of the motion compensated samples 108 a, 108 b, 108 c may representan image frame out of a series of consecutive image frames. In themotion compensated samples 108 a, 108 b, 108 c, a region of interest 68is present. Typically, the region of interest 68 comprises arepresentation of an indicative portion 12 of the subject 10 to bemonitored. Furthermore, each of the motion compensated samples 108 a,108 b, 108 c comprises a non-indicative motion region 110 which maybasically stand for a portion of the motion compensated samples 108 a,108 b, 108 c in which no indicative motion is expected. Consequently,initial motion (in non-compensated signal samples) and motion-relatedartifacts remaining after motion compensation (in the motion compensatedsamples) can be present in the non-indicative motion region 110.Preferably, the region of interest 68 is disregarded during motioncompensation accuracy assessment. As indicated above, it is preferredthat the minute characteristic motion pattern attributable to the vitalsignal of interest is preserved for signal extraction processing.Consequently, motion compensation accuracy assessment preferably is tobe based on at least a portion of the non-indicative motion region 110.

FIG. 7 a and FIG. 7 b illustrate similar approaches to motioncompensation accuracy assessment. Both approaches can make use of a sumof absolute difference algorithm applied to at least a portion of thenon-indicative motion region 110. In other words, a currentlyto-be-assessed motion compensated sample and a reference (motioncompensated) sample can be deduced from one another. A remainder, e.g.,a difference sample, obtained through this algorithm can be consideredindicative of motion compensation accuracy. Again, it is emphasized thatthe algorithm is merely applied to a region outside of the region ofinterest 68. In FIG. 7 a, the sum of absolute difference estimationoperation is indicated by reference numerals 112 a, 112 b. In FIG. 7 b,the sum of absolute difference estimation operation is indicated byreference numerals 114 a, 114 b. The above difference samples obtainedfrom the algorithm can form a basis upon which the evaluation parametercan be determined.

In FIG. 7 a a moving algorithm is applied to the motion compensatedsamples 108 a, 108 b, 108 c. That is, a currently to-be-assessed sampleis compared to a preceding reference sample. The distance or gap betweenthe to-be-assessed sample (e.g., 108 b) and the respective referencesample (e.g., 108 a) can be predefined and may be basically constant.The to-be-assessed sample and the respective reference sample may beadjacent or adjoining samples in the motion compensated sequence 106.However, it can be also envisaged that the to-be-assessed sample and therespective reference sample are spaced from one another in the motioncompensated sequence 106, that is, further samples can be interposedtherebetween. The alternative approach illustrated in FIG. 7 b makes useof a fixed reference sample (here the motion compensated sample 108 a).Consequently, each of a series of following consecutive motioncompensated samples 108 b, 108 c can be linked to the same referencesample 108 a. In this way, computational costs for motion compensationaccuracy assessment can be reduced. However, also a combination of themoving reference approach shown in FIG. 7 a and the fixed referenceshown in FIG. 7 b can be envisaged. In this way, a basically fixedreference sample can be updated (replaced by a new reference sample)periodically.

FIG. 8 illustrates another exemplary approach to motion compensationaccuracy assessment. In this embodiment, explicit featurecorrespondences in a to-be-processed sample (e.g., motion compensatedsample 108 b) and a reference sample (e.g., motion compensated sample108 a) can be detected. A number of detected feature correspondences,refer to correspondence lines 118 which have been added in FIG. 8 forillustrative purposes, may form a basis on which the evaluationparameter can be established. As mentioned above, it is preferred thatthe region of interest 68 is excluded from the detection of the featurecorrespondences 118. Consequently, features correspondence detection isto be applied to at least a portion of the non-indicative motion region110. Feature correspondence detection can make use of corner detection,edge detection, blob detection, ridge detection, etc. It goes withoutsaying that also the approach illustrated in FIG. 8 can make use of“moving” reference samples and “fixed” reference samples, refer to FIGS.7 a and 7 b.

FIG. 9 a and FIG. 9 b illustrate several exemplary signal forms 126, 128obtained through remote vital signal monitoring directed to thedetection of a subject's 10 respiration rate (or: breath rate). An axisof abscissas 122 denotes time (or, for instance, sample number, or framenumber) while an ordinate axis 124 denotes qualitative or quantitativesignal parameters representing detected motion. FIG. 9 a illustrates acharacteristic signal 126 which has been derived from a set of signalsamples to which no sufficient motion compensation measures have beenapplied. Consequently, the characteristic signal 126 is heavilycorrupted due to non-indicative motion disturbances. Consequently,applying vital signal extraction measures to the characteristic signal126 probably results in heavily distorted vital signal forms or values.It is therefore considered beneficial that, in accordance with someembodiments of the present invention, signal samples which are assessedand graded as “bad” samples can be excluded from downstream vital signalprocessing.

By contrast, FIG. 9 b illustrates a characteristic signal 128 whichclearly reflects an underlying at least partially periodic motionpattern which is attributable to indicative motion of the subject 10.The characteristic signal 128 has been derived from a set of motioncompensated samples which were graded as “good” samples. Disregardingbad samples in further signal processing may result in a characteristicsignal exhibiting an improved signal-to-noise ratio. Based on FIG. 9 b,for instance, an indicative frequency of recurring extreme values (e.g.,minima, maxima, etc.) can be detected which may represent the subject's10 respiration rate. It should be noted in this connection that thecharacteristic signal 128 may still comprise disturbances attributableto non-indicative motion. However, exemplarily referring to respirationrate determination, periodic changes in the characteristic signal 128are clearly visible and can therefore be processed and analyzed so as toderive the desired vital signal of interest.

Having demonstrated several alternative exemplary approaches covered bythe invention, FIG. 10 is referred to, schematically illustrating amethod for extracting information from remotely detected electromagneticradiation. Initially, in a step 150, an input data stream comprising asequence 152 of signal samples 154 a, 154 b, 154 c is received. An arrowt may denote time or an actual frame number. The data stream can bedelivered from a sensor means 22 or a data buffer or storage means. Thedata stream can be embodied by a series of image frames varying overtime.

A subsequent step 156 may provide for a determination of a region ofinterest 68 and, consequently, of a non-indicative motion region 110 inthe samples 154 a, 154 b, 154 c of the sequence 152. In a further step162, concurrently or lagged, motion compensation measures can be appliedto the samples 154 a, 154 b, 154 c so as to arrive at a sequence 158 ofmotion compensated samples 160 a, 160 b, 160 c. For instance, overallmotion can be addressed in this way which can be induced by sensormotion, or, specifically, camera motion, in particular with hand-heldmobile device applications. It is preferred in this connection that themotion compensation measures are based on motion detection which isbasically directed to a non-indicative region 110 in the signal samples154 a, 154 b, 154 c which is separate from the region of interest 68which is primarily addressed for extracting the desired vital signals.

A motion compensation quality assessment step 164 may follow. Again, insome embodiments, it is preferred that motion compensation qualityassessment is based on at least a portion of the non-indicative motionregion 110 in the motion compensated samples 160 a, 160 b, 160 c.Consequently, the region of interest 68 which assumingly exhibits anindicative motion pattern can be disregarded during the motioncompensation quality assessment operation. Depending on a detectedevaluation parameter 166 a, 166 b which may represent a grade of motioncompensation quality, the respective to-be-assessed sample can beexcluded from or included in further signal processing measures. Motioncompensation quality assessment may involve a comparative assessment ofa currently to-be-assessed sample 160 b, 160 c with respect to arespective reference sample 160 a and 160 b, respectively. The motioncompensation quality assessment operation 164 may further involve thegeneration of an output signal 168. The output signal 168 can bedirected to a user which can be advised to reduce disturbing motioninfluences, if required. In this connection, the user can be prompted tokeep the device 18 or, at least, the sensor means 22 stable. In additionto the desired indicative motion pattern the subject may have undesirednon-indicative body motion (e.g. may be moving his body to the left orright). Similar to camera motion the undesired non-indicative bodymotion may also cause disturbance for physiological informationextraction. The disturbance may be compensated for using the staticelements or objects in the background as reference, similar as describedabove. The output signal 168 directed to the user may provide advice toreduce disturbing motion influences caused by movement of the subject.

Motion compensated samples 160 a, 160 b, 160 c which are graded as“good” samples can form a signal basis from which a characteristicsignal 172 can be derived in a signal extraction step 170. Since heavilydistorted samples are excluded, the characteristic signal 172 canalready be considered highly indicative of the vital signal of interest.However, in some exemplary embodiments, a further signal enhancementoperation 174 may follow which may involve, for example, high passfiltering, low pass filtering, bandwidth filtering, windowing,statistical computation measures, etc. Consequently, an enhancedcharacteristic signal 176 can be computed. In yet another step 178,signal analysis measures can be applied to the enhanced characteristicsignal 176 or, in some cases, to the characteristic signal 172. Thesemeasures can be directed to seek for particular characteristicsindicative of at least one desired vital signal 180. Signal analysisoperation 178 may comprise transforming the characteristic signals 172,176 which are based in the time domain into a transformed signal whichis based in the frequency domain.

Needless to say, in an embodiment of a method in accordance with theinvention, several of the steps provided here can be carried out inchanged order, or even concurrently. Further, some of the steps could beskipped as well without departing from the scope of the invention. Thisapplies in particular to several alternative signal processing steps.

By way of example, the present invention can be applied in the field ofhealthcare, for instance, unobtrusive remote patient monitoring, generalsurveillances, securing monitoring and so-called lifestyle environments,such as fitness equipment, or the like. Applications may involvemonitoring of respiration rate, respiration rate variability and relatedvital signals.

While the invention has been illustrated and described in detail in thedrawings and the foregoing description, such illustration anddescription are to be considered illustrative or exemplary and notrestrictive; the invention is not limited to the disclosed embodiments.Other variations to the disclosed embodiments can be understood andeffected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims.

In the claims, the word “comprising” does not exclude other elements orsteps, and the indefinite article “a” or “an” does not exclude aplurality. A single element or other unit may fulfill the functions ofseveral items recited in the claims. The mere fact that certain measuresare recited in mutually different dependent claims does not indicatethat a combination of these measures cannot be used to advantage.

A computer program may be stored/distributed on a suitable medium, suchas an optical storage medium or a solid-state medium supplied togetherwith or as part of other hardware, but may also be distributed in otherforms, such as via the Internet or other wired or wirelesstelecommunication systems.

Any reference signs in the claims should not be construed as limitingthe scope.

1. A device for extracting physiological information from remotelydetected electromagnetic radiation emitted or reflected by a subject,comprising: an interface for receiving a data stream derivable fromelectromagnetic radiation emitted or reflected by a subject, the datastream comprising a sequence of signal samples including physiologicalinformation and indicative of disturbing motion, the signal samplesrepresenting at least one region of interest-exhibiting an at leastpartially periodic indicative pattern attributable to at least onephysiological parameter, and a non-indicative motion region; and aprocessing unit configured for processing the sequence of signalsamples, comprising; a stabilizing means configured for deriving asequence of derivative motion compensated samples at least partiallycompensated for undesired overall motion; a motion compensationassessment means configured for detecting an evaluation parameterrepresentative of motion compensation accuracy; and an extractor meansconfigured for selectively deriving at least one characteristic signalat least partially indicative of the at least partially periodicindicative pattern from the sequence of motion compensated samples,wherein deriving the characteristic signal is performed depending on thedetected evaluation parameter.
 2. A device as claimed in claim 1,wherein the signal samples are indicative of desired subject motion andof disturbing motion, the signal samples representing at least oneregion of interest exhibiting an at least partially periodic indicativemotion pattern, the characteristic signal being at least partiallyindicative of the at least partially periodic indicative motion pattern.3. A device as claimed in claim 1, wherein the processing unit furthercomprises an analyzing means configured for determining temporalvariations in the characteristic signal, the temporal variations beingrepresentative of at least one vital signal.
 4. A device as claimed inclaim 1, wherein the evaluation parameter is a flag parameterrepresentative of a state of a set of states indicative of motioncompensation accuracy for a given motion compensated sample, or a givenset of motion compensated samples.
 5. A device as claimed in claim 1,wherein the extractor means, on the basis of the actual evaluationparameter, selectively performs or omits processing the respectivemotion compensated samples for deriving the at least one characteristicsignal.
 6. A device as claimed in claim 1, wherein the stabilizing meansis configured for deriving the sequence of derivative motion compensatedsamples under consideration of at least one portion of thenon-indicative motion region in the signal samples.
 7. A device asclaimed in claim 6, wherein the stabilizing means is further configuredfor detecting and tracking local features in the least one portion ofthe non-indicative motion region in the signal samples of the sequence.8. A device as claimed in claim 1, wherein the motion compensationassessment means is configured for detecting a similarity between motioncompensated samples under consideration of at least one portion of thenon-indicative motion region in the motion compensated samples.
 9. Adevice as claimed in claim 8, wherein the motion compensation assessmentmeans is further configured for applying an absolute differenceprocessing algorithm to the at least one portion of the non-indicativemotion region in a respective motion compensated sample with respect toa reference sample.
 10. A device as claimed in claim 1, wherein themotion compensation assessment means is configured for detecting featurecorrespondences in at least one portion of the non-indicative motionregion in a respective motion compensated sample and in a referencesample, or wherein the motion compensation assessment means is furtherconfigured for detecting the evaluation parameter under consideration ofa plurality of motion compensation assessment indicators.
 11. A deviceas claimed in claim 1, further comprising a signal generation unitconfigured for generating a noticeable output signal depending on theactual evaluation parameter, wherein the output signal preferablyindicates a state of a set of states indicative of motion compensationaccuracy.
 12. A device as claimed in claim 1, further comprising asensor means, particularly a hand-held sensor means, configured forcapturing electromagnetic radiation within at least one particularwavelength range selected from the group consisting of visible light,infrared light, and ultraviolet radiation, the sensor means beingconnectable to the interface.
 13. A method for extracting physiologicalinformation from remotely detected electromagnetic radiation emitted orreflected by a subject, comprising: receiving a data stream derivablefrom electromagnetic radiation emitted or reflected by a subject, thedata stream comprising a sequence of signal samples comprisingphysiological information and indicative of disturbing motion, thesignal samples representing at least one region of interest exhibitingan at least partially periodic indicative pattern attributable to atleast one physiological parameter, and a non-indicative motion region;and processing the sequence of signal samples, comprising: deriving asequence of derivative motion compensated samples at least partiallycompensated for undesired overall motion; detecting an evaluationparameter representative of motion compensation accuracy; and derivingat least one characteristic signal at least partially indicative of theat least partially periodic indicative pattern from the sequence ofmotion compensated samples, wherein deriving the characteristic signalis performed depending on the detected evaluation parameter.
 14. Amethod as claimed in claim 13, wherein the signal samples are indicativeof desired subject motion and of disturbing motion, the signal samplesrepresenting at least one region of interest exhibiting an at leastpartially periodic indicative motion pattern, the characteristic signalbeing at least partially indicative of the at least partially periodicindicative motion pattern.
 15. A method as claimed in claim 13, furthercomprising the steps of: on the basis of the actual evaluationparameter, selectively performing or omitting processing the respectivemotion compensated samples for deriving the at least one characteristicsignal; and determining temporal variations in the characteristicsignal, the temporal variations being representative of at least onevital signal.